"The highlighted tokens are primarily morphemes, syllables, or word fragments in Hindi, Sanskrit, and related scripts, often marking grammatical, semantic, or phonetic units within words. These include suffixes, prefixes, conjuncts, and root components that are essential for word formation and meaning, as well as some full words or names in other Indic and East Asian languages. The activations focus on linguistically significant subword units that contribute to the structure and interpretation of complex words."
Score Type | Accuracy | Precision | Recall | F1 score | TPR | TNR | FPR | FNR |
---|---|---|---|---|---|---|---|---|
detection | 0.91 | 0.977 | 0.84 | 0.903 | 0.84 | 0.98 | 0.02 | 0.16 |
fuzz | 0.75 | 0.681 | 0.94 | 0.79 | 0.94 | 0.56 | 0.44 | 0.06 |